Log Barriers for Safe Black-box Optimization with Application to Safe Reinforcement Learning

Optimizing noisy functions online, when evaluating the objective requires experiments on a deployed system, is a crucial task arising in manufacturing, robotics and many others. Often, constraints on safe inputs are unknown ahead of time, and we only obtain noisy information, indicating how close we are to violating the constraints. Yet, safety must be guaranteed at all times, not only for the final output of the algorithm. We introduce a general approach for seeking a stationary point in high dimensional non-linear stochastic optimization problems in which maintaining safety during learning is crucial. Our approach called LB-SGD is based on applying stochastic gradient descent (SGD) with a carefully chosen adaptive step size to a logarithmic barrier approximation of the original problem. We provide a complete convergence analysis of non-convex, convex, and strongly-convex smooth constrained problems, with first-order and zeroth-order feedback. Our approach yields efficient updates and scales better with dimensionality compared to existing approaches. We empirically compare the sample complexity and the computational cost of our method with existing safe learning approaches. Beyond synthetic benchmarks, we demonstrate the effectiveness of our approach on minimizing constraint violation in policy search tasks in safe reinforcement learning (RL).Comment: 36 pages, 9 pages of appendi

Fungi in the Marine Environment: Open Questions and Unsolved Problems.

Terrestrial fungi play critical roles in nutrient cycling and food webs and can shape macroorganism communities as parasites and mutualists. Although estimates for the number of fungal species on the planet range from 1.5 to over 5 million, likely fewer than 10% of fungi have been identified so far. To date, a relatively small percentage of described species are associated with marine environments, with ∼1,100 species retrieved exclusively from the marine environment. Nevertheless, fungi have been found in nearly every marine habitat explored, from the surface of the ocean to kilometers below ocean sediments. Fungi are hypothesized to contribute to phytoplankton population cycles and the biological carbon pump and are active in the chemistry of marine sediments. Many fungi have been identified as commensals or pathogens of marine animals (e.g., corals and sponges), plants, and algae. Despite their varied roles, remarkably little is known about the diversity of this major branch of eukaryotic life in marine ecosystems or their ecological functions. This perspective emerges from a Marine Fungi Workshop held in May 2018 at the Marine Biological Laboratory in Woods Hole, MA. We present the state of knowledge as well as the multitude of open questions regarding the diversity and function of fungi in the marine biosphere and geochemical cycles

Evaluation of the kinase domain of c-KIT in canine cutaneous mast cell tumors

BACKGROUND: Mutations in the c-KIT proto-oncogene have been implicated in the progression of several neoplastic diseases, including gastrointestinal stromal tumors and mastocytosis in humans, and cutaneous mast cell tumors (MCTs) in canines. Mutations in human mastocytosis patients primarily occur in c-KIT exon 17, which encodes a portion of its kinase domain. In contrast, deletions and internal tandem duplication (ITD) mutations are found in the juxtamembrane domain of c-KIT in approximately 15% of canine MCTs. In addition, ITD c-KIT mutations are significantly associated with aberrant KIT protein localization in canine MCTs. However, some canine MCTs have aberrant KIT localization but lack ITD c-KIT mutations, suggesting that other mutations or other factors may be responsible for aberrant KIT localization in these tumors. METHODS: In order to characterize the prevalence of mutations in the phospho-transferase portion of c-KIT's kinase domain in canine MCTs exons 16–20 of 33 canine MCTs from 33 dogs were amplified and sequenced. Additionally, in order to determine if mutations in c-KIT exon 17 are responsible for aberrant KIT localization in MCTs that lack juxtamembrane domain c-KIT mutations, c-KIT exon 17 was amplified and sequenced from 18 canine MCTs that showed an aberrant KIT localization pattern but did not have ITD c-KIT mutations. RESULTS: No mutations or polymorphisms were identified in exons 16–20 of any of the MCTs examined. CONCLUSION: In conclusion, mutations in the phospho-transferase portion of c-KIT's kinase domain do not play an important role in the progression of canine cutaneous MCTs, or in the aberrant localization of KIT in canine MCTs

Topological Analysis of MAPK Cascade for Kinetic ErbB Signaling

Ligand-induced homo- and hetero-dimer formation of ErbB receptors results in different biological outcomes irrespective of recruitment and activation of similar effector proteins. Earlier experimental research indicated that cells expressing both EGFR (epidermal growth factor receptor) and the ErbB4 receptor (E1/4 cells) induced E1/4 cell-specific B-Raf activation and higher extracellular signal-regulated kinase (ERK) activation, followed by cellular transformation, than cells solely expressing EGFR (E1 cells) in Chinese hamster ovary (CHO) cells. Since our experimental data revealed the presence of positive feedback by ERK on upstream pathways, it was estimated that the cross-talk/feedback pathway structure of the Raf-MEK-ERK cascade might affect ERK activation dynamics in our cell system. To uncover the regulatory mechanism concerning the ERK dynamics, we used topological models and performed parameter estimation for all candidate structures that possessed ERK-mediated positive feedback regulation of Raf. The structure that reliably reproduced a series of experimental data regarding signal amplitude and duration of the signaling molecules was selected as a solution. We found that the pathway structure is characterized by ERK-mediated positive feedback regulation of B-Raf and B-Raf-mediated negative regulation of Raf-1. Steady-state analysis of the estimated structure indicated that the amplitude of Ras activity might critically affect ERK activity through ERK-B-Raf positive feedback coordination with sustained B-Raf activation in E1/4 cells. However, Rap1 that positively regulates B-Raf activity might be less effective concerning ERK and B-Raf activity. Furthermore, we investigated how such Ras activity in E1/4 cells can be regulated by EGFR/ErbB4 heterodimer-mediated signaling. From a sensitivity analysis of the detailed upstream model for Ras activation, we concluded that Ras activation dynamics is dominated by heterodimer-mediated signaling coordination with a large initial speed of dimerization when the concentration of the ErbB4 receptor is considerably high. Such characteristics of the signaling cause the preferential binding of the Grb2-SOS complex to heterodimer-mediated signaling molecules

PI3K inhibition results in enhanced HER signaling and acquired ERK dependency in HER2-overexpressing breast cancer

There is a strong rationale to therapeutically target the phosphatidylinositol 3-kinase/protein kinase B/mammalian target of rapamycin (PI3K/AKT/mTOR) pathway in breast cancer since it is highly deregulated in this disease and it also mediates resistance to anti-HER2 therapies. However, initial studies with rapalogs, allosteric inhibitors of mTORC1, have resulted in limited clinical efficacy probably due to the release of a negative regulatory feedback loop that triggers AKT and ERK signaling. Since activation of AKT occurs via PI3K, we decided to explore whether PI3K inhibitors prevent the activation of these compensatory pathways. Using HER2-overexpressing breast cancer cells as a model, we observed that PI3K inhibitors abolished AKT activation. However, PI3K inhibition resulted in a compensatory activation of the ERK signaling pathway. This enhanced ERK signaling occurred as a result of activation of HER family receptors as evidenced by induction of HER receptors dimerization and phosphorylation, increased expression of HER3 and binding of adaptor molecules to HER2 and HER3. The activation of ERK was prevented with either MEK inhibitors or anti-HER2 monoclonal antibodies and tyrosine kinase inhibitors. Combined administration of PI3K inhibitors with either HER2 or MEK inhibitors resulted in decreased proliferation, enhanced cell death and superior anti-tumor activity compared with single agent PI3K inhibitors. Our findings indicate that PI3K inhibition in HER2-overexpressing breast cancer activates a new compensatory pathway that results in ERK dependency. Combined anti-MEK or anti-HER2 therapy with PI3K inhibitors may be required in order to achieve optimal efficacy in HER2-overexpressing breast cancer. This approach warrants clinical evaluation

The pathology of familial breast cancer: The pathology of familial breast cancer How do the functions of BRCA1 and BRCA2 relate to breast tumour pathology?

Women with mutations in the breast cancer susceptibility genes, BRCA1 and BRCA2, have an increased risk of developing breast cancer. Both BRCA1 and BRCA2 are thought to be tumour suppressor genes since the wild type alleles of these genes are lost in tumours from heterozygous carriers. Several functions have been proposed for the proteins encoded by these genes which could explain their roles in tumour suppression. Both BRCA1 and BRCA2 have been suggested to have a role in transcriptional regulation and several potential BRCA1 target genes have been identified. The nature of these genes suggests that loss of BRCA1 could lead to inappropriate proliferation, consistent with the high mitotic grade of BRCA1-associated tumours. BRCA1 and BRCA2 have also been implicated in DNA repair and regulation of centrosome number. Loss of either of these functions would be expected to lead to chromosomal instability, which is observed in BRCA1 and BRCA2-associated tumours. Taken together, these studies give an insight into the pathogenesis of BRCA-associated tumours and will inform future therapeutic strategies

Dietary phytochemicals, HDAC inhibition, and DNA damage/repair defects in cancer cells

Genomic instability is a common feature of cancer etiology. This provides an avenue for therapeutic intervention, since cancer cells are more susceptible than normal cells to DNA damaging agents. However, there is growing evidence that the epigenetic mechanisms that impact DNA methylation and histone status also contribute to genomic instability. The DNA damage response, for example, is modulated by the acetylation status of histone and non-histone proteins, and by the opposing activities of histone acetyltransferase and histone deacetylase (HDAC) enzymes. Many HDACs overexpressed in cancer cells have been implicated in protecting such cells from genotoxic insults. Thus, HDAC inhibitors, in addition to unsilencing tumor suppressor genes, also can silence DNA repair pathways, inactivate non-histone proteins that are required for DNA stability, and induce reactive oxygen species and DNA double-strand breaks. This review summarizes how dietary phytochemicals that affect the epigenome also can trigger DNA damage and repair mechanisms. Where such data is available, examples are cited from studies in vitro and in vivo of polyphenols, organosulfur/organoselenium compounds, indoles, sesquiterpene lactones, and miscellaneous agents such as anacardic acid. Finally, by virtue of their genetic and epigenetic mechanisms, cancer chemopreventive agents are being redefined as chemo- or radio-sensitizers. A sustained DNA damage response coupled with insufficient repair may be a pivotal mechanism for apoptosis induction in cancer cells exposed to dietary phytochemicals. Future research, including appropriate clinical investigation, should clarify these emerging concepts in the context of both genetic and epigenetic mechanisms dysregulated in cancer, and the pros and cons of specific dietary intervention strategies

Safe Deep RL for Intraoperative Planning of Pedicle Screw Placement

Spinal fusion surgery requires highly accurate implantation of pedicle screw implants, which must be conducted in critical proximity to vital structures with a limited view of anatomy. Robotic surgery systems have been proposed to improve placement accuracy, however, state-of-the-art systems suffer from the limitations of open-loop approaches, as they follow traditional concepts of preoperative planning and intraoperative registration, without real-time recalculation of the surgical plan. In this paper, we propose an intraoperative planning approach for robotic spine surgery that leverages real-time observation for drill path planning based on Safe Deep Reinforcement Learning (DRL). The main contributions of our method are (1) the capability to guarantee safe actions by introducing an uncertainty-aware distance-based safety filter; and (2) the ability to compensate for incomplete intraoperative anatomical information, by encoding a-priori knowledge about anatomical structures with a network pre-trained on high-fidelity anatomical models. Planning quality was assessed by quantitative comparison with the gold standard (GS) drill planning. In experiments with 5 models derived from real magnetic resonance imaging (MRI) data, our approach was capable of achieving 90% bone penetration with respect to the GS while satisfying safety requirements, even under observation and motion uncertainty. To the best of our knowledge, our approach is the first safe DRL approach focusing on orthopedic surgeries.Comment: 10 pages, 4 figure